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Abstract 

The last decade has seen a remarkable development in the theory of 
asymptotics of Bayesian nonparametric procedures. Exponential con- 
sistency has played an important role in this area. It is known that 
the condition of /o being in the Kullback-Leibler support of the prior 
cannot ensure exponential consistency of posteriors. Many authors have 
obtained additional sufficient conditions for exponential consistency of 
posteriors, see, for instance, Schwartz [7], Barron, Schervish and Wasser- 
man [2] , Ghosal, Ghosh and Ramamoorthi [5 J , Walker [8j , Xing and Ran- 
neby [12]. However, given the Kullback-Leibler support condition, less 
is known about both necessary and sufficient conditions. In this paper 
we give one type of both necessary and sufficient conditions. As a con- 
sequence we derive a simple sufficient condition on Bayesian exponential 
consistency, which is weaker than the previous sufficient conditions. 
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1 Introduction 



Let X\,X2, ■ ■ ■ , X n be an independent identically distributed sample of n ran- 
dom variables, taking values in a Polish space X endowed with a cx-algebra 
X and having a common density / with respect to a dominated cr-finite 
measure n on X. For any two densities / and g, the Hellinger distance is 

1/2 



H(f,g) = ( /x(V f( x ) ~ V 9( x ) ) 2 / i (^ x )J an d the Kullback-Leibler diver- 
gence is K (/, gf) = J* x /(x) log fi(dx). Assume that the space F of densities 



is separable with respect to the Hellinger metric and assume that J- is the 
Borel cr-algebra of F. For a prior II on F, the posterior is the conditional 
distribution of II, given X\ % X2, ■ ■ ■ , X n , with the following expression 



U(A\X l ,X 2 ,...,X n ) 



n 



for measurable subsets A C F, where R n {f) = Yi {/(^i)//opQ)} stands 

i=i 

for the likelihood ratio. If the posterior Il(- | X\, X 2 , ... , X n ) concentrates on 
arbitrarily small neighborhoods of the true density /o almost surely or in prob- 
ability, then it is said to be consistent at /o almost surely or in probability 
respectively, where the almost sure convergence and the convergence in prob- 
ability are with respect to the infinite product distribution of the true 
distribution Pf with the density /o- The true density /o is said to be in the 
Kullback-Leibler support of the prior II if Il(/ : K(/o,/) < e) > for each 
e > 0. 

Consistency plays an important role in statistics. Early works on Bayesian 
nonparametric procedures were concerned with weak consistency of posteri- 
ors. Freedman [lj and Diaconis and Freedman [3] proved that a prior with 
positive mass on each weak neighborhood of /o cannot imply the weak consis- 
tency of posteriors. A sufficient condition on weak consistency was provided 
by Schwartz [7j, who proved that if /o is in the Kullback-Leibler support of II 
then the posteriors accumulate in all weak neighborhoods of /o- However, the 
Kullback-Leibler support condition is not enough to guarantee almost sure con- 
sistency of posteriors. Assume now that /o is in the Kullback-Leibler support 
of n. Barron, Schervish and Wasserman [2], Ghosal, Ghosh and Ramamoorthi 
[5], Walker [8], Xing and Ranneby [12] have obtained some sufficient condi- 
tions for posteriors to be almost surely consistent. The approaches of Barron 
et al. [2] and Ghosal et al. [5] are to construct suitable sieves and to compute 
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metric entropies. Their works were discussed in great detail in the monograph 
of Ghosh and Ramamoorthi [6], see also the nice review of Wasserman [9]. 
Walker's result [8] relies upon summability of squareroots of prior probability 
of suitable coverings. Xing and Ranneby [12] used the Hausdorff a-entropy 
to deal with the problem. In fact, all these almost sure consistency results 
are to establish sufficient conditions on exponential consistency of posteriors, 
i.e., posterior probabilities exponentially tend to zero. Much less is known 
about both necessary and sufficient conditions for exponential consistency of 
posteriors. To our knowledge there only exists a both necessary and sufficient 
condition due to Barron PQ, who used uniformly consistent tests to describe 
exponential consistency of posteriors. Barron's result has been widely applied 
in practice. In this paper we provide one type of both necessary and suffi- 
cient conditions for exponential consistency of posteriors. Our result shall be 
applied to give a verification of Barron's condition. As a consequence of our re- 
sults we obtain a sufficient condition for exponential consistency of posteriors, 
which implies several well known sufficient conditions. 

2 Consistency of Posterior Distributions 

In the section we give both necessary and sufficient conditions in the two 
senses: almost sure and in-probability. Some applications and consequences 
are discussed. 

We consider X n as a subset of X°° by identifying (x\,X2, ■ ■ ■ , x n ) G X n with 
the point [x\, X2, ■ ■ ■ , x n , 0, 0, . . . ) £ For a sequence {D^f of subsets 
D n C X", we denote 

limsup D n = {(xi, X2, . . . ) € X°° : (x%, X2, ■ ■ ■ , x n ) G D n infinitely often }. 

Barron pQ investigated relationship between exponential posterior consistency 
and existence of uniformly consistent tests. He obtained a characterization of 
exponential posterior consistency. Now we give one new type of characteriza- 
tions of exponential posterior consistency. 

Theorem 1. Suppose that the true density fo is in the Kullback-Leibler sup- 
port of 'II and that {An}^ be a sequence of subsets ofF. Then the following 
statements are equivalent. 

(i) There exists a constant (3$ > such that 

e nf3 °U(A n \X 1 ,X 2 ,...,X n ) -^0 

almost surely as n — > oo. 
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(ii) There exists a constant (3\ > such that 

P^{U(A n | X u X 2 , ■ ■ . , X n ) > e- n(Sl infinitely often } = 0. 

(iii) There exist constants < a\ < 1, fa > and a sequence {D n }J° of 
sets D n C X n with P?° (lira, sup D n ) = such that 




for all large n, 



where Ef stands for the expectation with respect to Xi,X%, . . . , X n 
and lx™\D n denotes the indicator function oflL n \D n . 

(iv) For each < a < 1 f/jere exist a constant f3 a > and a sequence 
{D n }f of sets D n C X" toitt P^(limsup D n ) = snc/i too* 

E fo (lx»\A, / ^(/) n ( d /)) a < e_n/3a for a11 lar S e «• 
J An 

(v) There exist a constant (3s > and a sequence {D n }^° of sets D n C X n 
suc/i i/iai P? 3 (lim sup D n ) = and 

f P / °°(X" \ D n ) H(4f) < e~ n/33 for all large n. 

J A n 

Note that for reader's convenience we include (ii) in Theorem[T]even though 
that the equivalence of (i) and (ii) is clear. 

Proof of Theorem [7J The implications (i) 4=> (ii) and (iv) =>■ (iii) are trivial. 
The equivalence (iii) 44> (v) follows directly from the equality 

%(V\D„/ Rn(f)Wfj)=[ E f0 {i xn \ Dn R n (f))n(df) 
= f Pf°(X n \D n )U(df). 

J A n 

So it suffices to prove (ii) =>■ (iv) and (iii) =>• (ii). To prove (ii) =>■ (iv), we set 

D„ = {(xi,x 2 ,...,x n ) € X n : n(A n |xi,x 2 ,...,x n ) > e~ n ^}. 
Then by (ii) we have Pj^(limsupD n ) = 0. Write 

A n = {feA n : Pf(X n \ J D n ) > e"^} U {f £ A n : Pf(X n \D n ) < e~^} 
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Given < a < 1, by the inequality (s + i) Q < s a + t a for s, t > we have 

Efo (J ai Rn(f)m)) a + E f0 (i X n XDn ^ i? n (/)n(d/)) a 



< 

It follows from Holder's inequality that 



Efo (f Ai Rn(f)Tl(df)) a < (E fo [J^ Rn(f)U(df)y' a y (E h li^) 1 Q 

= ( i {Ef.Rnif)) u(df)) a = n(d/))° < ^°°(x"\A0 n(#) 

0„ n 
TT/te) n(4r))M^i)---M<ten 



e 2 



( I it R ^f) n ( d /)) IT M X i) M^l) • • • M^n)") , 



which by the definition of D n does not exceed 



e ^- e -nf3 ia 



( I R n (f) U(df)) fl fo(xi) /i(dxi) . . . fi{dx n )] 

e-^f /(/ TT /(xj) /i(dxi) . . . n(dx n )) U(df)\ < . 



Similarly, we have 

Efo (ix»\s n / 2 i4(/)n(d/)) Q < (e }0 I i x „ XDn i? n (/)n(d/)) Q 

= ( / £ /o {l^ Dn R n (f))li(df)) a = ( / P/°(X"\ D n ) U(df)) a 

J A\ J A\ 

JVt- n W) ) 

Thus, we have obtained (iv) for /? Q = ^p. 



, a n f3i a 

< ( / e _ - 3~ n(df) < e — a-. 



Now we prove (iii) (ii). Note that 

n (A n | Xi,x%, x n ) 

= ^D n n(^4 n | Xi, X%, • • • , X n ^ + lx«\D n n(^4„ I Xi, X2, . . . , x n ) 

and 

P^(1d b / infinitely often) = (lim sup £>„) = 0. 
So for Pi = with the constants 02 and c-i from (iii), we have 

P^{H(A n \Xi,X 2 ,...,X n ) > e~ nPl infinitely often } 

= P£{lx»yj B n(A l I Xi, X 2 , . . . , X n ) > e-" A infinitely often } 

< P%[ljsn\ Dn f Rn(f)Wf) > e~ n2A infinitely often }, 

where the last inequality follows from 

P h { / R nU) n(#) < e~ nh infinitely often } = 0, 
see Lemma 4 of Barron et al. [2]. On the other hand, by (iii) we have 

J A n 
J A n 

< e n2 ^E f0 (l xn \ Dn I R n {f) U(df)) ai < e n2 ^~ n ^ = e~ n ^\ 

J An 

which by the first Borel-Cantelli Lemma yields that 

F%{lx»\D n [ Rn(f)mf) > e"™ 2/31 infinitely often } = 0. 

Thus we have obtain (ii) and the proof of Theorem Q] is complete. □ 

As an application of Theorem Q] we prove the following characterization 
of Barron [T], see also Theorem 4.4.3 in Ghosh and Ramamoorthi [5]. Recall 
that a test is a measurable function <p satisfying < <ft < 1. 
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Corollary 1. (Barron Suppose that the true density /q is in the 

Kullback-Leibler support of Ii and that {^4 n }^° be a sequence of subsets in 
¥. Then the following statements are equivalent. 

(i) There exists a constant (3q > such that 

e n ^U(A n \X 1 ,X 2 ,...,X n ) — >0 

almost surely as n — > oo. 

(ii) There exist subsets V n , W n ofF, positive constants c\, c 2 , (3\, /3 2 and 
a sequence of tests {<f> n = <p n (Xi, . . . , X n )} such that 

(a) A n cV n U W n ; 

(b) n(W n )<cie- n &; 

(c) Pft{<f>n > infinitely often} = and inf E f <j) n > 1 - c 2 e" n/32 . 

Proof. We need to prove that (ii) of Corollary [1] is equivalent to (v) of Theorem 
[Q Assume that (ii) holds. Set D n = {cf) n > 0}. Then P^(limsupZ) n ) = 
and 

/ P^(X n \D n )U(df)< [ Pf(X n \D n )U(df)+ [ Pf°(X n \D n )U(df) 
JA n ' Jv„ JW n 

<[ E f l Xn \ Dn U(df)+ f U(df)< ! E f {{l-<l> n )l X n\ Dn )U.{^) + cye 

JV n JWn JV„ 



n/9i 



< f c 2 e- nft U(df) + a e"^ 1 < c 2 e" n/32 + a e 
Jv„ 



which implies (v) for fa = (/?i A /? 2 )/2. Conversely, assume that (v) holds. So 
for (p n = lrj n we have that Pj°{(j) n > infinitely often} = Pj^limsup D n ) = 
0. Take W n = {/ E A n : P^°(X n \ D n ) > e"™^/ 2 } and V n = A n \ W n . Then 



n, 



e -r&/*U(W n ) < I P / °°(X n \ D n ) U(df) < e~ n ^ for all large 
which implies (b) for f}\ = /S3/2, and for each / G V n we have 

1 - p /( /> n = 1 - P]°{D n ) = pf (x°° \ AO = Pf(X" \ z? n ) < e - n ^/ 2 , 

which yields (c). Hence we have obtained (ii) and the proof of Corollary Q] is 
complete. □ 

Theorem [1] can be used to develop sufficient conditions for exponential 
posterior consistency. 
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Definition 1. Let d be a metric on F. The posterior distribution 
| X\, X2, ■ ■ ■ , X n ) is said to be exponentially consistent at the true den- 
sity /o almost surely (in probability) if for any e > there exists a constant 
(3 £ > such that 

e n & n(/ : d(f, f Q ) > e \ X u X 2 , . . . , X n ) — 

almost surely (in probability) as n — > 00. 

A direct consequence of Theorem [T] is the following result. 

Corollary 2. Let d be a metric on F and let r be a positive constant. Suppose 
that the true density /o is in the Kullback-Leibler support of II and suppose 
that for any e > there exist constants < a e < 1 and j3 £ > such that 

E fo ( [ R n (f) U(df)Y E < e~ n & for all large n. 

Then Il(- | Xx^X^ ■ ■ ■ ,X n ) is exponentially consistent at /o almost surely as 
n — > 00. 

Corollary [2] gives a sufficient condition for posterior consistency. It makes 
it possible to obtain posterior consistency without computation of metric en- 
tropies. In the following three corollaries we shall apply this sufficient condi- 
tion to verify the conditions given by Ghosal, Ghosh and Ramamoorthi [S], 
Walker [8] and Xing and Ranneby [12]. 

Let L^ be the space of all nonnegative integrable functions with the norm 
11/11 = C \f(x)\ fi(dx). Recall that the L^-metric entropy J(5,G) is the loga- 
rithm of the minimum of all numbers N such that there exist /1, /2, ■ ■ ■ , /jv i n 
L^ satisfying Q C UiLi{/ £ ^ : 11/ ~~ /ill < <^}> see Ghosal et al [5]. 

Corollary 3. (Ghosal et al |5j). Suppose that the true density function /o 
is in the Kullback-Leibler support of II and suppose that for any e > there 
exist < d < |, ci, C2 > 0, 0</3<^- ; and £/„ C F swc/i £/iai /or all large n, 

(a) n(F\g n ) <cie-" C2 ; 

(b) J(<5,0 ft ) <n/3. 
?7ien for any e > 0, 

n(/:||/-/o||>e|Xi,X 2 ,...,-Xn) ^0 
almost surely as n — > 00. 
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Proof. Given s > 0, by the inequality (s + t) 1 / 2 < s 1//2 + i 1 / 2 for s, t > we 
have 

1/2 



1/2 



V ^{/:||/-/o||>e} 

< E fo ( [ Rntf) U(df)) 1/2 + E f J ( R n (f) U(df) 

which, by Holder's inequality and (b), we have 

E f0 ( f R n (f)H(df)) 1/2 < (E f0 [ R n {f)Ii{df)) 112 
v Jv\g n J v J¥\g n J 

= ( [ E f0 R n (fMdf)) 1/2 = n(F\£ n ) 1/2 <c ie - nc ^ 2 . 
y Jw\g n ' 

On the other hand, by the proof of Theorem 2 in Ghosal et al [5], we know 
that (b) implies that there exist tests <f) n such that 

Efjn < e -"( £2 /8-/3) and inf Ef(j)n > 1 _ e -2n(e/i~&)\ 

/ee n :||/-/o||>£ 

Hence by Holder's inequality we get 

R n (f)IL(df)Y' 2 

'{feg n :\\f-f \\>e} 



EfJ [ Rn(f)Wf) 

v J{feg n :\\f-f \\>e} 

<E fo (4> n [ Rn(f)mf) 
V J{feg n :\\f-fo\\>s} 

+ E f J(l-4>n) [ Rn(f)U(df) 

v J{fee n :\\f-M\>s} 

< {Efjnf 2 (E f0 [ Rn(f)n(df) 

/ R n (f)n(df))) 

V V J{feg n :\\f-fo\\>e} ') 



1/2 



< e -n(eVW-m)f f II(d/) V '' 

1/2 



E f (l-<f> n )IL(df)) ' < e -n(eyW-m +e -n(e/4-5r 
{/efi»:||/-/o||>e} J 



+ 1 

Thus, using Corollary [21 we have proved Corollary [3l □ 



Corollary 4. (Walker |8j). Suppose that the true density /o is in the 
Kullback-Leibler support of H and suppsoe that for any e > there exist a 

covering {A\, A2, . . . ,Aj . . .} of {/ : H(f, /o) > e} and < 5 < e such that 

00 

£ y/ll(A]) < 00 and each A, C {/ : H(f jt f) < 5} for some density fj 
3=1 

satisfying H(fj, /o) > e. Then for any e > 0, 

U(f:H(f,f ) >e\X u X 2 ,...,X n ) ^0 
almost surely as n — > 00. 

Proo/. Denote = f A .f(x)R k {f)IL n (df)/J Aj R k (f)IL n (df) and 

P (/) = 1. Write 

/ p n (/) n n (d/ ) = u n (A 3 ) n /fc /; ( ^ fc+ ! ) - 

Then for any e > 0, by the inequality (s + i) 1 / 2 < s 1 / 2 + i 1 / 2 for s, t > 0, we 
have 



E f0 ( R n (f)n(dfj) <£%.( i?„(/)n(d/) x 

//b4,(^fc+l) X 1 / 2 



i=i 

n-l 



Hence, by Corollary [2j it is enough to show that there exists (3 > such that 
for all j and n, 

Using Jensen's inequality we have H(f k Aj, fj) 2 < <5 2 and hence H(fkAj, fo) > 
H(fj>fo) ~ H(fj,fkA ) > £ — <5 > 0. It then follows from Fubini's theorem 
that the last expectation is equal to 

/„ n— 2 

/ \J fn-iAj{x n )h{x n )n{dx n ) Y[ \J fkA^Xk+^foixk+iwidxi) . . . ^(dx n _i) 
x „_i x fc=o 

= / fl — 1 ^ J '^ 0, ' > ]J ^ f kAj (x k+1 )f (x k+1 )fi(dxi)...fi(dx n ^ 1 ) 
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< (1 2 ) J II y fkA 3 (x k+ i)fo(x k+1 ) fi(dxi) . . . flidXn-t) 

•4.1. / ( £ - 5 ) 2 \«/ r J £ -^) 2 

< . . . using the same argument • ■ • < (1 J < e 2 , 

which completes the proof of Corollary HI □ 

In Xing [10] [IT] and Xing and Ranneby [12j we developed an approach 
to estimate the expectation in Corollary where we used the Hausdorff a- 
entropy with the following definition. 



Definition 2. Let a > and ScFc L^. For 5 > 0, the Hausdorff a -entropy 
J(5, Q, a, II, d) of the set Q relative to the prior distribution U and the metric 
d is defined as 

N 

J(6,g,a,U,d) = log inf u ( B jT, 

3=1 

where the infimum is taken over all coverings {B\, B2, ■ ■ ■ , B^} of Q, where 
N may take 00, such that each Bj is contained in some ball {/ : d(f, fj) < 5} 
of radius 5 and center at fj G . 

Note that C(6, G, a, II, d) := e J ^ G ^ n ^ is called the Hausdorff a-constant 
of the subset Q. It was proved in [12] that for any < a < 1 and Q C F, 

c{5, g, a, n, d) < u{g) a n(s, g, df- a < n(s, g, d), 

where N(5, g, d) stands for the minimal number of balls of ci-radius 8 needed 
to cover g. Throughout this paper, by d we denote a metric such that it is 
bounded above by the Hellinger metric H and do(-,g) s is convex in F for some 
positive constant s and any g E F. For such a metric do we have 

Lemma 1. (Xing [llj). Let < a < 1, g C F and D re = {/ G g : 

do(f, fo) > re} with r > 2 and e > 0. Then we have 

Efo (J d Rn(f)H(df)) a < C(e,D r£ ,a,U,d )e^ r -^ ns2 . 

As a consequence of Corollary [2] we obtain the following result which es- 
sentially is Theorem 1 of Xing and Ranneby [12] . 



10 



Corollary 5. Let < a < 1 and (3\ > 0. Suppose that the true density fo 
is in the Kullback-Leibler support of II and suppose that for any e > there 
exist positive a constant f3 e > and a sequence of Q n C F such that 

(a) n(F\g„)<e-^; 

(b) C(e,g n ,a,U,d ) <e n ^\ 
Then for any £ > 0, 

U(f:d (fJ )>e\X 1 ,X 2 ,...,X n )^0 

almost surely as n — > oo. 

Note that for a = 1 we have C(e,Q n ,a,H,do) = Tl(Gn) < 1 which yields 
(b) of Corollary [5j Hence Corollary [5] does not hold when a = 1. 

Proof of Corollary O Assume that r is a large positive constant which will be 
determined later. For any e > 0, by the inequality (s + t) a < s a + t a for 
s, t > we have 

EfA ! R n (f)U(df)Y 

<E fo ( [ R n {f)Tl{df)) a + E f0 ( / R n (f)n(df)Y, 
which, by Holder's inequality and Lemma [H does not exceed 

E fo [ R n (f)n(df)Y + C(e,Gn,a,U,d )e^^ 2ns2 
Jw\g n ' 

< ( [ E fQ R n (fMdf)Y + ^+^(r~^ 

v JF\g n ' 

Take r so large that (3\ + (r — 2) 2 < — /3 e . Then we have 

£/ f / R n (f)U(df)Y <e~^ for all large n, 

V J{f--d (fJo)>re} ' 

which by Corollary [2] completes the proof of Corollary [5j □ 

Finally, we present a both necessary and sufficient theorem for in- 
probability exponential consistency of posteriors, which is an analogue of The- 
orem [TJ 
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Theorem 2. Suppose that the true density /q is in the Kullback-Leibler sup- 
port of II and that {A n }]° be a sequence of subsets o/F. Then the following 
statements are equivalent. 

(i) There exists a constant (3q > such that 

e n(3o U(A n \X 1 ,X 2 ,...,X n ) — >0 

in probability as n — > oo. 

(ii) There exists a constant 0\ > such that 

P%{ll(A n \X 1 ,X 2 ,...,X n ) >e~ n ^} — >0 as n ^ oo. 

(iii) There exist constants < ai < 1, (3 2 > and a sequence {D n }^ of 
sets D n C X n with Pj°{D n ) — ► as n — > oo suc/i i/iat 

(V\i?n ^ ^(/) n(d/)) ai < e"^ 2 for all large n. 

(iv) For eac/i < a < 1 i/iere exist a constant f3 a > and a sequence 
{D n }f of sets D n C X n with P^{D n ) as n -> oo suc/i tfia* 

£/o (lx»\A, / ^(/) n (#))" < e" n/3a for all large n. 
J A n 

(v) There exist a constant fa > and a sequence {-D n }f° o/ sets D n C X n 
suc/i i/iai PJ^{D n ) — > as n — > oo and 

/ P/°(X n \ D n ) n(d/) < e- n& for all large n. 
Ja„ 

The proof of Theorem [2] follows essentially from the same lines as the proof 
of Theorem Q] and therefore is omitted. Similar to the proof of Corollary Q3 we 
get a result of Barron's type on in-probability posterior convergency. 

Corollary 6. Suppose that the true density /o is in the Kullback-Leibler sup- 
port ofH and that {A n }^° be a sequence of subsets in F. Then the following 
statements are equivalent. 

(i) There exists a constant fa > such that 

e n(3 °U(A n \X 1 ,X 2 ,...,X n ) — >0 

in probability as n — > oo. 

(ii) There exist subsets V n , W n o/F, positive constants c\, c 2 , 0\, fa and 
a sequence of tests {(j) n = <p n (Xi, . . . , X n )} such that 

(a) A n cV n UW n ; 

(b) n(W n ) <cie-"fr; 

(c) P^{0n > 0} -> as n ->■ oo and inf Ef<p n > 1 - c 2 e" n/32 . 
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